Online panels offer the benefits of lower costs, timeliness and access to large respondent numbers compared to surveys that use more traditional modes of data collection (face-to-face, telephone, and mail). Probability-based online panels in which potential respondents are selected randomly from a frame covering the target population allow for drawing valid conclusions about the population of interest. With the increase in nonresponse over the last years in surveys using the traditional modes of data collection (de Leeuw & de Heer, 2002), probability-based online panels are an appealing alternative. However, whether the quality of data collected by probability-based online panels is comparable to the quality of data attained by traditional data collection methods is questionable. Probability-based online panels are expensive in their recruitment and maintenance, thereby raising the question: are the costs of constructing and maintaining them justified given the quality of data that can be obtained?

There are several sources of errors in online panel surveys: excluding non-Internet users may result in coverage error; if persons selected for the study cannot be reached or do not want to participate, it may result in nonresponse error; study participants may choose to stop participating in later waves (attrition). Furthermore, by taking surveys regularly, respondents can learn to answer dishonestly or answer filter questions negatively to reduce the burden of participation (panel conditioning). All these errors can accumulate in the survey estimates and the conclusions based on these estimates may therefore be misleading.

The goal of this dissertation is to study the quality of data obtained with probability-based online panels. The dissertation aims at advancing the understanding of the causes of errors and at guiding the design decisions when recruiting and maintaining probability-based online panels. This dissertation evaluates the overall quality of estimates from an online panel and focuses on potential sources of error: nonresponse during the recruitment interview, nonresponse to the first online survey, panel attrition, panel conditioning, and the effects of the survey mode.

This dissertation consists of five studies, which are theoretically integrated by the overarching framework of the Total Survey Error (Biemer, 2010; Groves, 1989; Groves & Lyberg, 2010). The framework is extended by including theoretical knowledge on two special types of panel survey errors – panel nonresponse (attrition) and panel conditioning (Kalton, Kasprzyk, & McMillen, 1989). The empirical analyses are based on the data from a probability-based telephone-recruited online panel of Internet users in Germany: the GESIS Online Panel Pilot.

The error sources are studied in connection to the recruitment and operating steps typical for probability-based online panels. Each chapter studies a different aspect of data quality. The chapters are written as individual papers, each addressing specific research questions.

Chapter 1 introduces the theoretical framework of data quality and the Total Survey Error. The goal of Chapter 2 is to evaluate the goodness of the final estimates collected in the online panel. The data from the online panel are compared to the data from two high-quality face-to-face reference surveys: the German General Social Survey “ALLBUS” and the German sample of the European Social Survey (ESS). Furthermore, since researchers may not only be interested in single estimates, I study to what extent survey results are comparable when the data are used for modeling social phenomena. The results show several differences among the surveys on most of the socio-demographic and attitudinal variables; however, these differences average to a few percentage points. To account for the design decision to exclude non-Internet users, post-survey adjustments were performed. However, post-stratification weighting did not bring the estimates from the online panel closer to the estimates from the reference surveys.

Chapter 3 focuses on nonresponse, studying the influence of respondent characteristics, design features (incentives and fieldwork agency), and respondent characteristics specific to the survey mode (Internet experience, online survey experience). The results indicate that participation in the panel is selective: previous experience with the Internet and online surveys predicts willingness to participate and actual participation in the panel. Incentives and fieldwork agencies that performed the recruitment also influence the decision to participate.

Chapter 4 studies why panel members choose to stay in the panel or discontinue participation. The main question is whether respondents are motivated by intrinsic factors (survey experience) or extrinsic factors (monetary incentives). The findings indicate that respondents who view surveys as long, difficult, too personal are likely to attrite and that incentives (although negatively related to attrition) do not compensate for this burdensome experience.

Chapter 5 focuses on panel conditioning due to learning the survey process. To find out if more experienced respondents answer differently than less experienced respondents, I conducted two experiments, in which the order of the questionnaires was switched. The findings indicate limited evidence of advantageous panel conditioning and no evidence of disadvantageous panel conditioning.

Chapter 6 studies mode system effects (i.e., differences in the estimates as the result of the whole process by which they were collected). The data from the online panel is compared to the two reference surveys: ALLBUS 2010 and ALLBUS 2012. Both face-to-face surveys were recruited by the same fieldwork agency and employed an almost identical design. Therefore, they together serve as a “reference mode” when compared with the online panel. I use questions with identical wordings that were present in both ALLBUS surveys and replicated in the online panel. The differences in sample composition among the surveys are adjusted by propensity-score weighting. The results show that the online panel and two reference surveys differ in attitudinal measures. However, for factual questions the reference surveys differ from the online panel and not from each other. Judging by the effect sizes, the magnitude of the differences is small. The overall conclusion is that data from the online panel is fairly comparable to the data from high-quality face-to-face surveys.

The results of this dissertation provide additional insight into the processes that contribute to the quality of data produced by probability-based online panels. These results can guide researchers who plan to build online panels of Internet users or of the general population. The results will also prove useful for existing panels that consider switching to the online mode.